In this project we attempted to utilize these components …(list out all the components and images here).. Each of which will be discussed in further detail in the following sections.
To tackle the problem mentioned in the objective section, three different biomedical sensors are tested using different methods and integrated into the system we designed. The three sensors are high accuracy temperature sensor, GSR sensor and pulse sensor, and their data are sent to an Arduino Uno. Then, the Uno transmits the data to Raspberry Pi 4. We intended to extract data from these biomedical sensors in real-time and then to do a decent judgment whether the person is lying or not. In order to show a straightforward result in the final demo, a simple game of guessing ball in cups was implemented and a dashboard of showing the nerves level was implemented both using the pygame library.
Polygraph Test Credit: Gorodenkoff/Shutterstock
System Overview
Project Objective:
A polygraph, commonly known as a lie detector test, has been used as a way to tell if a person tells a lie or not. This project is to build an embedded system using raspberry pi and biomedical sensors to simulate such a device which can sense and detect whether the user is telling the truth or not. A simple game is also to be designed to demonstrate the result and judgment made by the system.
Design and Testing
Figure 3. Temperature Sensor, GSR Sensor and Pulse Sensor
Temperature Sensor
The temperature sensor worked perfectly fine with a high accuracy of 0.0001, and it communicated using I2C. We observed an issue with the temperature sensor, which is that the sensor has its own hardware temperature and usually starts with room temperature. It takes a while(usually 2-3 mins) to get heat equilibrium with the person’s finger. So, if the system wants to monitor the person’s body temperature and extract some pattern, it is not ready to use in a short time, which may affect the overall user experience of the system.
Establish Bluetooth Connection
The bluetooth component of the project involved communication between the Pi Zero and the Pi 4. The original intention is to hook up all the sensors onto the Pi Zero and communicate the data to the Pi 4 to achieve the concept of a lightweight wearable. This didn’t end up working out due to sensor integration reasons that will be discussed in the next section. Here, we will talk about successfully establishing the connection between the Zero and 4 via Bluetooth.
The Bluetooth module is needed to first be activated with the ‘rfkill unblock Bluetooth’ command, similar to what we did in Lab1 for WIFI. The status could be then confirmed with ‘systemctl status bluetooth’ as shown in the figure below. The command ‘hciconfig’ gives us the bd_addr, which is the bluetooth module identification address that will be needed in the python scripts to directly establish a RFCOMM connection.
Figure 4. RFCOMM Connection
On the Pi 4 as the server, we open up the Bluetooth socket and listen for communication to a specified port. For the Pi Zero as the client, the script initiates the connection with the given bd_addr and port number. Once the connection is established, data can be sent between the two devices. We modified the sample code below to keep the connection open with a while loop with the client sending commands to tell the server script what to do next.
GSR Hat to connect GSR sensor to Pi Zero
We ordered a hat for the GSR sensor because it uses an analog signal, which is not acceptable by RaspberryPi. The hat is also made by the same company as the GSR sensor, and theoretically, RPi can get correct data through the hat from the GSR sensor. We ran into lots of problems when we tried to install the software for the hat according to the provided tutorial. We tried the simple one-click installation at first but it didn’t work. Then, I tried a detailed step by step installation, but we were stuck at one of the dependency installations - Install MRAA & UPM. Prof and TA helped us debug and provided different ideas but still cannot fix it. As the GSR data is the most important data in this project that reflects human’s biomedical status, we decided to utilize Arduino Uno to collect all data from three sensors and to send the data to Pi4 using the serial port.
Figure 5. Grove Hat
Figure 6. Sensor Data on Arduino Serial Monitor
The GSR sensor worked well with Arduino and the communication between Arduino and Pi4 was also successfully built, which set all sensor data ready for use by the Pi4. The GSR sensor basically measures the conductivity of the skin. If a person gets nervous, he usually will sweat a little, which will result in the increase of skin conductivity. We first looked at the serial plotter in the Arduino IDE and tried to observe any patterns or features. The GSR data fluctuates a lot and we can only see the overall level change, so we decided to average the GSR data first in the Arduino code and care more about the overall level.
Arduino Uno to read sensor data and communicate with Pi 4
Figure 7 is the hardware diagram of the how the sensors are connected to the Arduino Uno. The maxrefdes117 pulse-oximeter is powered by the 3V port with the I2C pins connected to the arduino I2C ports and the interrupt connected to pin 10. The GSR sensor is powered by the 5V port and sends its data through the A0 analog port.
Figure 7. Arduino Connection Layout
Figure 7 is the hardware diagram of the how the sensors are connected to the Arduino Uno. The maxrefdes117 pulse-oximeter is powered by the 3V port with the I2C pins connected to the arduino I2C ports and the interrupt connected to pin 10. The GSR sensor is powered by the 5V port and sends its data through the A0 analog port.
The next key part is to get communication working between the Arduino and the Pi4, especially for Pi4 reading data from Arduino using serial communication. We set up hardware permission for serial communication and installed the python serial library for Pi. The Arduino side platform would print the data through serial and the Pi 4 would read the serial line using the python serial library.
Figure 8. Serial Communication Python Example
A test script for GSR level was written to observe the difference of GSR level when a person tells the truth and lies. For the first 20 secs, the person will be asked to speak out letters and numbers on the keyboard that are pointed and he must answer correctly. The GSR data average will be recorded and calculated. To be more specific, we use a sliding window average, which stores the average data from Arduino to a list and we calculate the last 20 elements in the list. Then, we will use these averages to calculate the final average for each 20 secs in both the calibration and testing stage. For another 20 secs, he will be asked the same questions but he must answer incorrectly. Also, the GSR data average will be recorded and calculated. A significant difference between two averages was observed, which will be mentioned in the results section.
We noticed that the absolute value of the sensor data varied between trials, thus we couldn’t simply compare the current value with a fixed baseline. Noticing that lie detection is more about the change of state, we decided to implement a calibration stage allowing us to measure the baseline of the user at the specific moment then compare the questioning measurements with said baseline.
The last sensor is the heart rate sensor. It turned out the sensor we received was just the sensor itself and not integrated onto a board for direct connection using jumper cables. We were able to borrow the maxrefdes117 integrated component from Sheryas to continue our work. We successfully extract heart rate data using Arduino code and send it to the Pi. The heart rate data had two outputs with the first one being the heart rate and the second one being the valid bit (Fig. 6). When testing, the heart rate sensor can reflect the basic level of heart rate according to whether a person is staying still calmly or jumping up and down, but it cannot differentiate when a person lies because 1) quick lies don’t cause that drastic of a change in heart rate 2) the computed heart rate fluctuates too much to confidently detect any slight changes in the heart rate
Game design and integration
Our goal is to use the percentage change from the baseline of these three sensors to calculate a stress score that determines the emotional shift of the user. We start by evaluating the three sensors separately to see their individual responses to the change in the user (i.e. lying in this case) to determine the weight each sensor should have on the overall stress score. With the characteristics of the sensors defined, we can represent and combine the data into an equation as shown. We could then integrate this score into our cup game and use it to determine the end results.
Equation 1: Stress score calculation equation
As previously discussed, we noticed that the temperature sensor does not effectively capture helpful change so we decided not to include its data in the equation. We weighed GSR data more heavily since it is the most responsive to the lying condition. We weighed the heart rate significantly lower due to our observation of its inconsistent result despite no change in user status.
It is very tricky to do a proper demo game design to show our testing results. The general idea of the game is to divide the game into two parts. The first part is to set a baseline by asking some normal questions such as ‘Is the sky blue?’ The baseline data shows the person’s biomedical status when he is telling the truth. For the testing part, the baseline data can be used to judge whether he is lying or not.
We designed a cup game. The game process is that the player will answer a series of common sense questions correctly in the calibration stage. Then, he will choose one cup from four cups to hide the ball in. After that, he will be asked whether the ball is in a particular cup or not five times for a single cup. The player must answer no to all questions, which means that he will lie when he is asked if the ball is in the cup he chose previously.
Figure 9. Game Process
Result
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum lorem nulla, consectetur at leo vel, pretium bibendum nisl. Cras blandit quam a enim ultrices, eu convallis enim posuere. Donec eleifend enim sed purus consectetur, vitae cursus lectus varius. Vivamus consectetur felis nec est venenatis posuere. Phasellus vitae aliquet erat. In laoreet lacinia mollis. Quisque iaculis nisl fermentum pharetra lobortis. Donec rhoncus dui sem, ac molestie leo tristique vel. Phasellus in nibh feugiat, fringilla lectus in, elementum magna. Etiam quis dui condimentum, tempus ex in, dapibus est. Cras ut congue augue. Donec ac enim ex. Ut id tristique risus, vel porttitor quam. Sed ultricies enim eu nibh porttitor, vel sodales enim feugiat. Fusce volutpat venenatis magna ac ultrices. Curabitur eget urna ut nulla mattis convallis non eu diam.
A baseline data will be calculated during the calibration stage. Four averages will also be recorded and calculated for four different cups when the player is asked questions. An overall score can be generated by the data from GSR sensor and heart rate sensor. A judgment will be made by comparing the baseline average and four different cup question averages. According to the test we made, the lower the score is, the more possible the ball is in that cup.
The overall results left much to be desired. We envisioned a light-weight wearable capable of differentiating emotional status and detecting any lie. We hoped to demonstrate the effectiveness of our sensor data by incorporating the data in the form of a “find the ball” cup game implemented through pygame. We were able to successfully create the pygame interface that incorporated an overall game flow from calibration, in-game questioning, result display and show a person’s emotional status through visual representation in the form of a stress meter
Figure 10. Stress Score Monitoring Screen
The issues we came across could be roughly categorized into two sections: 1. Human Variability 2.Sensor consistency
Human Variability
The core concept of our project predicated on the sensor being able to pick up the autonomic arousal of the user. This required us to first experiment what conditions could trigger noticeable change in the sensor reading.
We found out that human biometric status does not vary significantly with isolated yes/no questions as there is neither enough weight behind the question to trigger a significant emotional response nor a long enough exposure time for the sensor to pick up the changes. When considering how to ask the questions, we realize that an open-ended format (i.e. describing a true and false story) potentially introduced many human factors that cannot be quantified, thus we chose a more straightforward approach in asking the user to identify the keys on a keyboard (i.e. first truthfully, then falsely). With the “describe a true/false story” calibration method, we saw a significant difference in GSR readings with the false story showing a lower reading. The more controlled calibration method (i.e. keyboard key identification) showed results differentiating between ±10-20 in GSR reading (Fig. 10), which was also a significant difference considering that these results are the average of the sliding window measurements.
Figure 11. GSR Calibration Test (keyboard keys identification)
We noticed that our questioning options were limited in game (e.g. asking which cup) which made it more difficult to trigger a significant response, while accounting for other human factors such as anticipation and engagement. Using repetition of the same question, we were able to lengthen the duration of any particular state that the user was in and allow the sensor to differentiate truth and lie. This was determined heavily by the GSR as we noticed that heart rate data doesn’t change noticeably. Figure 11 is the result screen of one of the test runs where we chose cup 1 to hide the ball in. We can see that the GSR reading for cup 1 does indeed appear to be the lowest. However, we notice that the difference is not as significant with only a ±5 in reading.
Figure 12. Cups game Result Screen
Sensor Consistency
Going back to figure 11, the calibration GSR value is much lower than all the other values. We’ve noticed that throughout our various experiments the sensors are extremely sensitive to any changes in the environment. For example, if we were to put our hands on the ESD pad while running measurements this would drastically shift the outputting values. Another example is if we use the fingers on the same hand to activate both the GSR sensor and pulse-oximeter, the measurements will become skewed (i.e. this is verified by lifting the finger on the pulse-oximeter off). As for the pulse-oximeter, any slight pressure change or shift of the finger could cause a change in the measurements. We can see from figure 12that the heart rate fluctuates between two 15 second intervals. This is why we impose a strict calibration check (i.e. requiring two consecutive measurements to be within ±10) before moving onto the biometric dashboard. We observed that there are a lot of factors that could cause interference with the reading. Averaging the results from the sensor helped stabilize the sensor output to a degree where we could observe an overall trend in rise (i.e. heart rate increase from jumping up and down) or fall (i.e. GSR drop from user lying) that is due to the user and not ambient interference.
Our project achieves the capability of sensing the heart rate and GSR changes stimulated from physical exertion and long duration lying. Although the resolution of such detection is under significant external interferences, we were able to mitigate some of the interference with various averaging methods and weight calculation.
Overall, we believe that there were still a lot of options to explore in order to create a more accurate and stable lie detector as the results were still fairly inconsistent. The obstacles we’ve encountered illustrated the difficulty of embedded systems design and integration due to the continuous update of the Pi systems. Our experience with this project showed that it is not impossible to design a much more consistent lie detector but much more factors would need to be taken into consideration.
I believe we relied too heavily on the arduino library for the pulse-oximeter which had an algorithm for calculating the heart rate. We were trying to come up with a questioning scheme and tinkering with the weight contribution for each sensor but it’s generally not a good approach to try and make a broken system work. Although it was a bit too late, we realized that the reason the sensor results (i.e. specifically the heart rate data) was so jumpy might be due to the lack of memory of the Arduino Uno board. Therefore, the limited sampling points used to compute the Fourier transform to obtain the heart rate were not sufficient in providing a consistent result despite the user remaining still.
Future Work
As mentioned in the conclusion, we would like to implement our own signal processing algorithm from scratch. Instead of doing the processing on the Uno board, we would transmit the raw data from the pulse-oximeter to the Pi 4. Making use of the amazing computing power and memory of the Pi 4 along with the SciPy library, we can calculate a more consistent heart rate via the discrete Fourier transform of a much larger sampling pool. On top of that, we could implement a filtering system that takes care of the interferences caused by users shifting their finger.
On the questioning scheme side of things, we could definitely work on a more well-rounded method that extracts as much human factors as possible. One thought is we should’ve randomized the cup at which the computer chooses to ask to minimize the effects of human anticipation. A more extensive exploration would be to implement voice recognition. This allows us to go beyond yes/no questions and even possibilities of analyzing the confidence of one’s voice.
Diagram
Figure 14. Game Flowchart
Work Distribution
Brian Hsu
hh543@cornell.edu
Pygame design and integration with pulse-oximeter sensor, Data processing of sensor data, Bluetooth communication, Game flow design
Yitong Sun
ys555@cornell.edu
Polygraph paper research, Data processing of sensor data, Arduino communication,, Game flow design.
